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Scientific  and  Technical  Objectives 


Our  objective  has  been  to  produce  a  general  method  for  controller  development  that  requires 
minimal  domain  expertise  and  development  time,  yet  can  outperform  traditional  control  methods 
in  several  applications  relevant  to  the  Navy’s  strategic  objectives.  To  demonstrate  the  utility  of 
the  approach,  a  high  degree-of-freedom  (DOF),  three-dimensional  (3D)  biped  has  been 
incorporated  as  a  testbed  system. 


Approach 

The  approach  is  to  develop  and  refine  algorithms  in  simulation,  with  an  emphasis  on  the 
difficulties  associated  with  transfer  to  hardware. 

Approach  1:  Our  control  algorithms  are  developed  to  ensure  that  they  work  in  hardware, 
exploiting  naturalistic  locomotor  movements  and  passive  dynamics.  This  is  a  key  difference 
between  our  work  and  previous  research,  which  has  focused  almost  exclusively  on  simulation 
results.  We  have  constructed  a  nine  DOF  biped  robot  to  test  the  ability  of  algorithms  to 
overcome  problems  including  sensor  noise,  actuator  variability,  parts  wear,  variable  friction, 
time  delays,  nonlinear  interaction  forces,  and  difficult-to-model  dynamics. 

Approach  2:  We  test  candidate  control  algorithms  on  reduced  problems  that  permit  us  to  quickly 
explore  the  effects  of  changing  evolution  operators  and  parameters.  This  ensures  numerical 
tractability  while  exploring  issues  of  scalability,  computational  speed,  and  adaptability  to 
nonstationary  dynamics.  For  this  purpose,  two-dimensional  (2D)  five-link  and  seven-link 
walking  models  have  been  developed. 

Approach  3:  We  have  developed  complete  dynamic  simulations  of  the  biped  in  the  ADAMS  and 
Open  Dynamics  Engine  (ODE)  physics  simulation  environments.  Training  the  controller 
primarily  in  simulation  as  opposed  to  the  actual  robot  permits  the  controller  to  learn  from 
thousands  of  walking  trials  and  reduces  wear  on  the  mechanical  parts.  A  ‘gray  box’  modeling 
approach  is  used  to  integrate  data  gathered  during  walking  trials  on  the  actual  biped  to  adaptively 
improve  the  ODE  model. 


Concise  Accomplishments 

We  have  achieved  the  following  during  the  current  reporting  period: 

1.  Completed  coding  and  testing  of  a  2D,  seven-link  model  (Figure  1).  Significance:  the 
seven-link  model  now  incorporates  feet,  and  hence  the  capability  for  ankle  push-off  and 
locomotion  over  rough  terrain. 


2.  Implemented  the  neuroevolution  algorithm  on  the  .2D,  seven-link  model  for  stable 
walking  over  rough  terrain  (Figure  2).  Significance:  The  controller  framework  is  now 
fully  developed  and  ready  for  implementation  on  a  3D  walking  model. 

3.  Implemented  neuroevolution-based  gait  initiation  and  gait  termination  on  the  seven-link 
model.  Significance:  when  applied  to  the  robot,  it  should  be  able  to  start  and  stop 
autonomously,  removing  the  need  for  difficult  and  unreliable  human  “launches.” 

4.  Completed  development  of  a  3D  ODE  simulation  of  the  complete  biped  (Figure  3). 
Significance:  The  ODE  simulation  is  necessary  to  train  the  final  controller  before  transfer 
to  hardware. 

Expanded  Accomplishments 

In  the  past  year,  we  have: 

1.  Completed  coding  and  testing  of  a  2D,  seven-link  model.  The  model  consists  of  an 
upper  body,  two  thigh  segments,  two  shin  segments,  and  two  feet.  The  terrain  is  simply 
modeled  as  a  sequence  of  (x,  y)  points.  Interpenetration  constraints  and  joint-limits  are 
enforced  using  soft  constraints,  and  friction  is  modeled  as  traditional  Coulomb  friction. 
Coded  in  C-H-,  the  model  runs  extremely  efficiently  -  about  40x  faster  than  real  time  on 
an  Intel  Core  2  Duo  T7300  laptop  processor  (including  the  control  code).  This  speed  is 
critical  as  it  allows  learning  algorithms  to  be  tested  quickly  on  available  hardware. 

2.  Implemented  the  neuroevolution  algorithm  on  the  2D,  seven-link  model  for  stable 
walking  over  rough  terrain.  In  moving  from  the  five-link  to  the  seven-link  model,  it 
was  discovered  that  the  non-discrete  double-stance  phase  required  implementation  of  a 
state  machine  for  successful  walking.  The  state  machine  consists  of  two  possible  states: 
‘push-off  and  ‘swing,’  which  is  sent  to  the  neurocontroller  as  a  binary  variable.  In  this 
framework,  a  critical  part  of  learning  to  walk  involves  deciding  whether  the  current  state 
is  ‘push-off  or  ‘swing’  -  a  decision  that  depends  strongly  on  the  nature  of  the  local 
terrain,  e.g.  smooth  downhill/uphill  or  abrupt  step-down/step-up. 

The  desired  task  for  the  seven-link  model  was  stable  and  efficient  locomotion 
over  rough  terrain.  The  terrain  height  varied  each  step  with  a  standard  deviation  of  6% 
the  biped’s  leg  length  (see  Figure  1).  The  neuroevolution  was  run  for  1000  generations 
15  times,  each  run  producing  a  stable  and  efficient  controller.  Upon  the  completion  of 
each  run,  the  final  controller  was  tested  for  1(X)  “test  walks,”  each  lasting  until  1000 
Joules  of  energy  are  expended  or  the  biped  falls  down.  On  the  most  successful  run,  the 
biped  fell  on  only  5%  of  test  walks  (traveling  an  average  of  67  meters,  ~103x  its  leg 
length)  and  displaying  a  Specific  Cost  of  Transport  (a  normalized  measure  of  energy 
efficiency)  of  0.142.  By  comparison,  for  humans  walking  on  level  ground  this  value  is 
about  0.2,  and  for  Honda’s  Asimo  it  is  3.23.  A  video  of  the  resulting  gait  can  be 
downloaded  from  here: 

http://www.mech.northwestem.edu/hartmann/Neuroevolved_Biped_Videos.html. 


3.  Implemented  neuroevolution-based  gait  initiation  and  gait  termination  on  the  seven- 
link  model.  The  limit  cycle  walking  controller  currently  instantiated  on  the  robot 
requires  a  human  to  manually  “launch”  (push)  it  into  a  stable  limit  cycle.  This  is 
unfavorable,  due  to  both  the  obvious  need  for  autonomous  gait  initiation  for  any  useful 
bipedal  robot,  and  the  unreliability  of  the  procedure  for  even  an  experienced  human.  To 
address  this  shortcoming,  the  above  seven-link  model  experiments  incorporated  a  gait 
initiation  component  with  the  locomotion  controller,  with  the  biped  starting  at  rest  and  its 
legs  side-by-side.  The  evolved  controllers  all  displayed  smooth  and  reliable  gait 
initiation  behavior,  a  video  a  where  can  be  downloaded  from  here: 
http://www.mech.northwestem.edu/hartmann/Neuroevolved  Biped  Videos.html 

In  addition,  a  gait  termination  component  was  also  easily  evolved.  Aside  from 
the  clear  usefulness  of  gait  initiation/termination,  the  straightforwardness  with  which  the 
procedure  was  implemented  indicates  that  the  neuroevolution  approach  can  also  be 
extended  to  subsume  additional  locomotion  capabilities,  such  as  turning,  walking 
sideways  and  backwards,  avoiding  obstacles  and  more. 

4.  Completed  development  of  a  3D  ODE  simulation  of  the  complete  biped.  Compared 
to  the  existing  ADAMS  model,  it  is  geared  more  towards  computational  efficiency  and 
low-level  programmability. 
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